Learning About and From Variation in Program Impacts Using Multi-site Trials Working Draft

The present paper, which is intended for a diverse audience of evaluation researchers, applied social scientists and research funders, provides a broad overview of the conceptual and statistical issues involved in using multi-site randomized trials to learn about and from variation in program effects across individuals, across policy-relevant and theoretically-relevant subgroups of individuals and across program sites. Learning about variation in program effects involves detecting and quantifying this variation. Learning from variation in program effects involves studying the factors which predict or explain it. The paper has four parts plus a brief final discussion. Part One introduces the concepts and issues involved. Part Two focuses on detecting and quantifying variation in effects of program assignment, which are often referred to as effects of intent to treat (ITT). Part Three extends the discussion to variation in effects of program participation, which are often referred to as a complier average causal effect (CACE) or a local average treatment effect (LATE). Part Four considers moderators of program effects (individuallevel or site-level factors that influence the sign and magnitude of these effects) and mediators of program effects (individual-level or site-level “mechanisms” by which these effects are produced.